1. Field of the Invention
The present invention relates to sophisticated interfaces between humans and machines. More particularly, the invention concerns a method and apparatus for determining when a subject has eye contact with a specific area.
2. Description of the Related Art
The phrase xe2x80x9cuser friendlinessxe2x80x9d refers to a machine""s ease of use by a human being. For every machine, there may be some people that do not consider it user friendly, and therefore object to using it. As a notorious example, thousands of videocassette recorder (VCR) clocks across America idly blink the time of xe2x80x9c12:00xe2x80x9d because of the perceived difficulty of setting them. Clearly, increasing the power of the man-machine interface is an important step toward unlocking the potential of our machines.
Many existing tools enable humans to make their wishes known to machines. In simpler cases, there are levers, knobs, wheels, dials, pushbuttons, cranks, and the like. With more sophisticated machines, such as computers, there are mechanically operated input devices such as joysticks, mice, keyboards, digitizing pads, foot pedals, and the like. Moreover, engineers have developed even more complicated and powerful devices and techniques, such as voice control, eye gaze tracking, motion sensors, etc. Although these systems constitute a significant advance and enjoy widespread commercial success today, the engineers and scientists at International Business Machines Corp. (IBM) continually seek improvements in the performance and efficiency of human-machine interfaces.
Broadly, the present invention concerns a method and apparatus to determine when a subject has eye contact with a specific target area. In one embodiment, the invention accomplishes this by estimating an angle between (1) the direction in which the subject is looking and (2) the direction from the subject directly to the target area. This technique accesses whether the subject is looking in a particular direction.
In one embodiment, the angle is estimated using the position of a glint of light in the subject""s pupil. A light source, which may be positioned within the target area to simplify computations, directs light toward the eye to create a glint upon the subject""s cornea. A light sensor, positioned within the target area, then detects the glint and measures the glint position. The invention also generates or otherwise receives a machine-readable representation of the pupil. This representation may comprise, for instance, identification of certain pixels that represent the pupil in a camera image.
The glint position is then analyzed with respect to the pupil to determine whether their relative positions meet certain criteria. In one embodiment, this may involve determining whether the glint position is sufficiently central to the pupil. Accordingly, if the glint position is sufficiently central, and the light source and camera are positioned near the target area, the invention decides that the subject is looking at the target area.
According to another embodiment, the invention may use face orientation rather than glint position to determine whether the subject is looking in the target area. In this embodiment, the invention first generates or otherwise receives a machine-readable input representing the three-dimensional position of multiple points on the subject""s face. As an example, these points may represent the tip of the nose, the nostrils, the corners of the mouth, and the subject""s eyes. Using these points, the invention computes a face plane describing an average orientation of the face. A refined estimate can be obtained using the relative position of the eyes in their eye sockets.
Next, the orientation of the face plane is analyzed with respect to the target area. As an example, this may be performed by computing the dot product of the face plane and the vector between the face and the target area. Only if the face plane is sufficiently normal to this vector does the invention decide that the subject is looking at the target area.
A particularly useful aspect of the invention conditions eye contact estimations according to the subject""s distance from the target area, because there is less tolerance for divergent angles when the subject is farther away. This conditioning helps to more accurately determine whether the subject is looking at a particular target object. Range between the subject and the target object may be further used to determine whether to use the glint-based embodiment or the face-plane-embodiment. The glint-based embodiment is better for closer ranges, whereas the face-plane embodiment is advantageous for farther ranges where glint may be imperceptible. Thus, the invention may be implemented to use the glint-based embodiment unless long range makes it difficult to distinguish glint from pupil, whereupon the face plane embodiment is employed.
The invention may be implemented in various embodiments. In one embodiment, the invention may be implemented to provide a method to determine when a subject is looking at a specific area. In another embodiment, the invention may be implemented to provide a logic circuit or a computer-driven apparatus for determining when a subject is looking at a specific area. In still another embodiment, the invention may be implemented to provide a signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital data processing apparatus to perform operations for determining when a subject is looking at a specific area.
The invention affords its users with a number of distinct advantages. First, the invention facilitates more natural human-machine interaction by determining when a subject is looking at a designated area, such as the machine itself. The invention also helps avoid potential errors by using eye contact to determine the user""s intent when issuing a potentially ambiguous command in an environment populated with multiple voice operated machines. The invention also provides a number of other advantages and benefits, which should be apparent from the following description of the invention.